Method for intercepting specific system calls in a specific application from applications space for security

ABSTRACT

One or more specified system calls of a running process are trapped in kernel space from user space. While the process is stopped, information associated with the process is read and a security analysis is performed on the information to determine whether malicious code activity is detected, such as a buffer overflow. If malicious code activity is detected, protective action is taken, such as killing the specified system call. Otherwise, if malicious code activity is not detected, the specified system call is restarted.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to the protection of computer systems.More particularly, the present invention relates to a method forintercepting calls in a running process from user space to allowsecurity analysis to be performed.

2. Description of Related Art

Most operating systems currently provide debugging support, such asthrough special debugging APIs (Application Program Interfaces).Typically, a debugging API allows a software developer to halt theexecution of a running program from user space and to examine the stateof the process, such as the variables and the stack, in order toidentify and correct programming errors in the program code.

SUMMARY OF THE INVENTION

In accordance with one embodiment of the invention, a method includesestablishing a break upon entry into one or more specified system callsof a process running in user space, wherein the break stops the one ormore specified system calls in kernel space prior to execution. Upon thebreak, information associated with the process is dynamically read, anda security analysis is performed on the information to determine whethermalicious code activity is detected.

If malicious code activity is detected, protective action is taken, suchas killing the specified system call. Otherwise, if malicious codeactivity is not detected, the specified system call is restarted.Embodiments in accordance with the present invention can be used toprotect privileged processes from malicious code attacks, such as bufferoverflow attacks or return to LIBC attacks.

Embodiments in accordance with the present invention are best understoodby reference to the following detailed description when read inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a client-server system that includes a securityapplication executing on a host computer system in accordance with oneembodiment of the present invention.

FIG. 2 illustrates a flow diagram of a process implemented by thesecurity application of FIG. 1 in accordance with one embodiment of theinvention.

FIG. 3 illustrates a block diagram of a /proc file being modified tobreak for a system call in accordance with one embodiment of theinvention.

FIG. 4 illustrates a block diagram of a poll being set in accordancewith one embodiment of the invention.

Common reference numerals are used throughout the drawings and detaileddescription to indicate like elements.

DETAILED DESCRIPTION

Embodiments in accordance with the present invention utilize applicationdebugging features of an operating system to trap a specified systemcall(s) in the kernel space from user space and to perform a securityanalysis on information associated with a parent process of thespecified system call(s) for the presence of malicious code activity. Inparticular, the present invention utilizes the application debuggingfeature provided by many operating systems that allows the applicationsdebugger to break on, also termed herein trap, a system call from userspace.

For purposes of description, the present invention is described withreference to a Solaris operating system and the /proc file system.However, the present invention is not limited to the present example,and can be used on other operating systems on which an applicationsdebugger can be run. For example, gdb on Unix, Linux and Windowsoperating systems traps system calls in the kernel space. The /proc filesystem is similar on Solaris and AIX operating systems, but otheroperating system platforms may require a port.

With reference to Solaris, generally, the /proc file system is a filesystem that provides access to the state of each process running on theSolaris operating system and serves as a debugging API from user space.Herein the term user space refers to application space that is externalto the kernel space. In the /proc file system, each running process isidentified using a PID (process identifier). Debuggers, the /proc filesystem, user space, and kernel space are all terms well-known to thoseof skill in the art and so are only briefly discussed to avoiddetracting from the principles of the invention.

Referring generally, to FIG. 2, in one embodiment, a method includesopening a /proc file for a specified process that is running in userspace (operation 204) and modifying the /proc file for the specifiedprocess to break, i.e., stop execution, on entry to a specified systemcall(s) (operation 206). A poll is established on the /proc file for thespecified (operation 208) to alert when the polls are set so thatinformation associated with the specified process can be read once thespecified process is stopped. When the specified process is stopped onentry to the specified system call(s) (operation 210), the informationassociated with the specified process is read (operation 212) andanalyzed to determine whether malicious code is detected in the process(operation 214). Upon a determination that malicious code activity isdetected, protective action is taken (operation 218), such as killing oraborting the trapped system call. Optionally, a notification of theprotective action taken is provided (operation 220). Otherwise, if adetermination is made that malicious code activity is not detected, thetrapped system call is restarted (operation 216).

In particular, FIG. 1 is a diagram of a client-server system 100 thatincludes a security application 106 executing on a host computer system102 in accordance with one embodiment of the present invention. Hostcomputer system 102, sometimes called a client or user device, typicallyincludes a central processing unit (CPU) 108, hereinafter processor 108,an operating system 104, such as Solaris, an input/output (I/O)interface 110, and a memory 114. In the present embodiment, operatingsystem 104 includes a debugging interface which can break on a systemcall of a running process from user space, for example, a /proc filesystem debugging API.

Host computer system 102 may further include standard devices like akeyboard 116, a mouse 118, a printer 120, and a display device 122, aswell as, one or more standard input/output (I/O) devices 124, such as acompact disk (CD) or DVD drive, floppy disk drive, or other digital orwaveform port for inputting data to and outputting data from hostcomputer system 102. In one embodiment, security application 106 isloaded into host computer system 102 via I/O device 124, such as from aCD, DVD or floppy disk containing security application 106. In oneembodiment, security application 106 is run as a daemon application.

In the present illustration, host computer system 102 is coupled to aserver computer system 130 by a network 126. Server computer system 130typically includes a display device 132, a processor 134, a memory 136,and a network interface 138. Network 126 can be any network or networksystem that is of interest to a user. In various embodiments, networkinterfaces 138 and I/O interface 110 include analog modems, digitalmodems, or a network interface card.

In the present embodiment, security application 106 is stored in memory114 of host computer system 102 and executed on host computer system102. The particular type and configuration of host computer system 102and server computer system 130 are not essential to this embodiment ofthe present invention.

More particularly, FIG. 2 illustrates a flow diagram of a process 200implemented by security application 106 in accordance with oneembodiment of the invention. Referring now to FIGS. 1 and 2 together,execution of security application 106 by processor 108 results in theoperations of process 200, in which, in one embodiment, a specifiedprocess that is running on host computer system 102 is located bysecurity application 106 from user space, and process 200 is entered atENTER operation 202.

In one embodiment, the specified process is a privileged process, suchas telnetd, however, other processes which are desirable to protect frommalicious code activity can also be specified. For purposes ofdescription it is assumed that the PID assigned this specified processis 250. From ENTER operation 202, processing transitions to an OPEN/PROC FILE operation 204.

In OPEN /PROC FILE operation 204, the /proc file for the specifiedrunning process, e.g., /proc/250, is opened and processing transitionsfrom OPEN /PROC FILE operation 204 to a MODIFY CTL FILE operation 206.

In MODIFY CTL FILE operation 206, the ctl file of the /proc file for thespecified process, e.g., /proc/250/ctl, is modified to break on aspecified system call(s). The ctl file is a write-only file to whichcontrol messages can be written to direct the operating system to changesome aspect of the process' state or control its behavior.

To break on a system call during a running process, typically rootpermission is required. In one embodiment, a PCSENTRY structure iswritten to the /proc/<pid>/ctl file of the specified process. ThePCSENTRY command message instructs a process to stop on entry to, i.e.,break on, a specified system call(s). This command message changes the/proc ctl file and modifies the system calls for that process.

For example, in one embodiment, the /proc/250/ctl file is modified bywriting /proc/250/ctl FORK and ENTRY for exec such that process 250 willstop on entry to the system call to exec, and the system call will betrapped in the kernel space prior to execution. In some embodiments, aPCSET with PR_FORK can be written to the /proc/<pid>/ctl so that all thechildren inherit the same debug flags and trap on the same system calls.From MODIFY CTL FILE operation 206, processing transitions to anESTABLISH POLL operation 208.

In ESTABLISH POLL operation 208, in one embodiment, a poll isestablished on the /proc file of the specified system call. In oneembodiment, the poll is set to alert security application 106 when thefile descriptors POLLWRNORM and POLLPRI are set for the /proc/<pid>ctlfile. Processing waits, e.g., sleeps, until the specified system call isinitiated, and from ESTABLISH POLL operation 208, processing transitionsto a STOP PROCESS operation 210.

In STOP PROCESS operation 210, when a break on the specified process ismade on entry to the specified system call, e.g., exec, the specifiedprocess is stopped. From STOP PROCESS operation 210, processingtransitions to a READ DATA operation 212.

In READ DATA operation 212, once the process is stopped and the pollconditions set, e.g., POLLWRNORM and POLLPRI are set and securityapplication 106 is alerted, data is read from selected /proc files ofthe specified process, such as the /proc/<pid>/status and/proc/<pid>/info files, e.g., /proc/250/status and /proc/250/info files.The following is an example of information available from the statusfile about a specified process:

typedef struct pstatus { int pr_flags; /* flags (see below) */ intpr_nwlp; /* number of lwps in the process */ pid_t pr_pid; /* process id*/ pid_t pr_ppid; /* parent process id */ pid_t pr_pgid; /* processgroup id */ pid_t pr_sid; /* session id */ id_t pr_aslwpid; /* lwp-id ofthe aslwp, if any */ id_t pr_agentid; /* lwp-id of the agent lwp, if any*/ sigset_t pr_sigpend; /* set of process pending signals */ uintptr_tpr_brkbase; /* virtual address of the process heap*/ size_t pr_brksize;/* size of the process heap, in bytes */ uintptr_t pr_stkbase; /*virtual address of the process stack */ size_t pr_stksize; /* size ofthe process stack, in bytes */ timestruc_t pr_utime; /* process user cputime */ timestruc_t pr_stime; /* process system cpu time */ timestruc_tpr_cutime; /* sum of children's user times */ timestruc_t pr_cstime; /*sum of children's system times */ sigset_t pr_sigtrace; /* set of tracedsignals */ fltset_t pr_flttrace; /* set of traced faults */ sysset_tpr_sysentry; /* set of system calls traced on entry */ sysset_tpr_sysexit; /* set of system calls traced on exit */ lwpstatus_t pr_lwp;/* status of the representative lwp */ } pstatus_t;

Additionally, the folllowing is an example of information available foreach thread of a specified process:

typedef struct lwpstatus { int pr_flags; /* flags (see below) */ id_tpr_lwpid; /* specific lwp identifier */ short pr_why; /* reason for lwpstop, if stopped */ short pr_what; /* more detailed reason */ shortpr_cursig; /* current signal, if any */ siginfo_t pr_info; /* infoassociated with signal or fault */ sigset_t pr_lwppend; /* set ofsignals pending to the lwp */ sigset_t pr_lwphold; /* set of signalsblocked by the lwp */ struct pr_action; /* signal action for currentsignal */ sigaction stack_t pr_altstack; /* alternate signal stack info*/ uintptr_t pr_(—) /* address of previous ucontext */ oldcontext; shortpr_syscall; /* system call number (if in syscall) */ short pr_nsysarg;/* number of arguments to this syscall */ int pr_errno; /* errno forfailed syscall */ long pr_sysarg /* arguments to this syscall */[PRSYSARGS]; long pr_rval1; /* primary syscall return value */ longpr_rval2; /* second syscall return value, if any */ char pr_clname /*scheduling class name */ [PRCLSZ]; timestruc_t pr_tstamp; /* real-timetime stamp of stop */ u_long pr_instr; /* current instruction */prgregset_t pr_reg; /* general registers */ prfpregset_t pr_fpreg; /*floating-point registers */ } lwpstatus_t;

Further, the following is an example of information available from theinfo file about a specified process:

typedef struct psinfo { int pr_flag; /* process flags */ int pr_nlwp; /*number of lwps in the process */ pid_t pr_pid; /* process id */ pid_tpr_ppid; /* process id of parent */ pid_t pr_pgid; /* process id ofprocess group leader */ pid_t pr_sid; /* session id */ uid_t pr_uid; /*real user id */ uid_t pr_euid; /* effective user id */ gid_t pr_gid; /*real group id */ gid_t pr_egid; /* effective group id */ uintptr_tpr_addr; /* address of process */ size_t pr_size; /* size of processimage in Kbytes */ size_t pr_rssize; /* resident set size in Kbytes */dev_t pr_ttydev; /* controlling tty device (or PRNODEV) */ u_shortpr_pctcpu; /* % of recent cpu time used by all lwps */ u_shortpr_pctmem; /* % of system memory used by process */ timestruc_tpr_start; /* process start time, from the epoch */ timestruc_t pr_time;/* cpu time for this process */ timestruc_t pr_ctime; /* cpu time forreaped children */ char pr_fname /* name of exec'ed file */ [PRFNSZ];char pr_psargs /* initial characters of arg list */ [PRARGSZ]; intpr_wstat; /* if zombie, the wait ( ) status */ int pr_argc; /* initialargument count */ uintptr_t pr_argv; /* address of initial argumentvector */ uintptr_t pr_envp; /* address of initial environment vector */lwpsinfo_t pr_lwp; /* information for representative lwp */ } psinfo_t;typedef struct lwpsinfo { int pr_flag; /* lwp flags */ id_t pr_lwpid; /*lwp id */ uintptr_t pr_addr; /* internal address of lwp */ uintptr_tpr_wchan; /* wait addr for sleeping lwp */ char pr_stype; /*synchronization event type */ char pr_state; /* numeric lwp state */char pr_sname; /* printable character for pr_state */ char pr_nice; /*nice for cpu usage */ short pr_syscall; /* system call number (if insyscall) */ char pr_oldpri; /* pre-SVR4, low value is high priority */char pr_cpu; /* pre-SVR4, cpu usage for scheduling */ int pr_pri; /*priority, high value = high priority */ u_short pr_pctcpu; /* % ofrecent cpu time used by this lwp */ timestruc_t pr_start; /* lwp starttime, from the epoch */ timestruc_t pr_time; /* cpu time for this lwp */char pr_clname /* scheduling class name */ [PRCLSZ]; char pr_name /*name of system lwp */ [PRFNSZ]; processorid_(—) pr_onpro; /* processorwhich last ran this lwp */ t processorid_(—) pr_bindpro; /* processor towhich lwp is bound */ t psetid_t pr_bindpset; /* processor set to whichlwp is bound */ } lwpsinfo_t;

Once the data is read, from READ DATA operation 212, processingtransitions to a MALICIOUS CODE DETECTED check operation 214.

In MALICIOUS CODE DETECTED check operation 214, a security analysis isperformed utilizing information read from the /proc file system inoperation 210, such as the /proc files status and info, as well as other/proc files. In one embodiment, a security analysis is performed on theinformation read during operation 210 to determine whether maliciouscode activity is detected. For example, in one embodiment, stackinformation is analyzed for buffer overflow as the stack should not havechanged during a system call. Additionally, in one embodiment, returnfrom LIBC attacks can be detected by opening the /proc/<pid>/objectfiles and the preceding command from the return can be checked todetermine if it is a call. Further, the return address can also bemapped to ensure it is associated with a mapped page.

Upon a determination that malicious code activity is detected (“YES”),processing transitions from MALICIOUS CODE DETECTED check operation 214to a PROTECTIVE ACTION operation 218.

In PROTECTIVE ACTION operation 218, protective action is taken, such asblocking completion of the trapped system call. For example, in someembodiments, PCKILL or PCSABORT (abort system call) are written to the/proc ctl file or the process can be left in a STOP state for furtheranalysis. From PROTECTIVE ACTION operation 218, processing optionallytransitions to a NOTIFY operation 220, or directly to an EXIT operation222 if optional NOTIFY operation 220 is not performed.

In optional NOTIFY operation 220, a notification of the protectiveaction taken is provided to a user or other recipient, such as a systemadministrator. From optional NOTIFY operation 220, processingtransitions to EXIT operation 222, with processing exiting method 200.

Referring again to MALICIOUS CODE DETECTED check operation 214, upon adetermination that malicious code is not detected (“NO”), processingtransitions from MALICIOUS CODE DETECTED check operation 214 to aRESTART operation 216.

In RESTART operation 216, the trapped system call, e.g., exec, isrestarted, and the system call is allowed to complete. For example, theprocess can be restarted by writing a PCRUN to the /proc/<pid>/ctl file.From RESTART operation 216, processing transitions to EXIT operation222, with processing exiting method 200.

In an alternative embodiment, a notification can also be received beforethe system call exits, by writing PCSEXIT to the /proc ctl file.

FIG. 3 illustrates a block diagram of a /proc ctl file being modified tobreak for a system call of process 250 in accordance with one embodimentof the invention. In FIG. 3, at block 302 security application 106, fromuser space, writes a modification to the /proc ctl file of runningprocess PID 250 (identified at block 308) to break at a system call toexec. At block 304, the /proc ctl file is modified to break at thespecified system call, e.g., exec, represented by block 306 as anuncompleted system call, pre_syscall.

FIG. 4 illustrates a block diagram of a poll being met for the /procfile of process 250 in accordance with one embodiment of the invention.In FIG. 4, at block 408 the running process PID 250 initiates an ftpd atblock 410 in user land that enters a system call for exec in the kernelspace, e.g., presyscall, at block 406. As the /proc ctl file was earliermodified to break at the entry to the system call and a poll placed onthe /proc file, at block 404, when the poll conditions are set, e.g.,POLLWRNORM and POLLPRI are set, security application 106 is alerted andinformation associated with the process 250 is read by securityapplication 106 and used for determining whether malicious code activityis detected.

Method 200 provides one embodiment of the present invention wherespecified system calls in a running process are controlled from userspace. In particular, specified processes are stopped from running whentrapped on the execute system call.

Other desired security features can also be included in accordance withthe present invention. For example, a process can be stopped fromremoving a file, and one particular process can be stopped from makingone particular system call having specified parameters. Additionally,all children of a process can be set to be monitored. If inetd ismonitored, control can be gained over all the network system calls on acomputer system, providing a simple NIDS (Network Intrusion DetectionSystem) solution.

Further, using nice, a process can be throttled dynamically by raisingand lowering its priority. Using syscall., the system call number juststopped on can be read. Using argc, the number of arguments to thesystem call can be read. Using argv, the value of arguments to thesystem call can be read. Using envp, the environment vector can be read.Using stkbase and stksize, the status of the stack before and after aparticular system call can be read (return from LIBC). Using brkbase andbrksize, the status of the heap can be read. Using pr_reg[R_PC], thereturn address can be checked to check for a buffer or heap overflow.Using pr_reg[R_SP], the current stack pointer can be checked (returnlibc).

Thus, as described herein, in accordance with the invention, specifiedsystem calls of specified processes are trapped in the kernel space fromuser space. Information associated with the process is read and asecurity analysis is performed on the information to determine whethermalicious code activity is detected, such as a buffer overflow. Ifmalicious code activity is detected, protective action is taken, such askilling the specified system call. Otherwise, if malicious code activityis not detected, the specified system call is restarted.

Embodiments in accordance with the invention permit other user spacesystem call interception to be by-passed. Further, embodiments inaccordance with the invention work independent of kernel space patchlevels and are portable to other operating systems. The binary of theprocess does need to be modified, and the target processor does not needto be recompiled.

Referring again to FIG. 1, security application 106 is in computermemory 114. As used herein, a computer memory refers to a volatilememory, a non-volatile memory, or a combination of the two.

Although security application 106 is referred to as an application, thisis illustrative only. Security application 106 should be capable ofbeing called from an application or the operating system.

In one embodiment, an application is generally defined to be anyexecutable code. Moreover, those of skill in the art will understandthat when it is said that an application or an operation takes someaction, the action is the result of executing one or more instructionsby a processor.

While embodiments in accordance with the present invention have beendescribed for a client-server configuration, an embodiment of thepresent invention may be carried out using any suitable means and/orhardware configuration involving a personal computer, a workstation, aportable device, or a network of computer devices. Other networkconfigurations other than client-server configurations, e.g.,peer-to-peer, web-based, intranet, internet network configurations, areused in other embodiments.

Herein, a computer program product comprises a medium configured tostore or transport computer readable code in accordance with anembodiment of the present invention. Some examples of computer programproducts are CD-ROM discs, DVDs, ROM cards, floppy discs, magnetictapes, computer hard drives, servers on a network and signalstransmitted over a network representing computer readable code.

As illustrated in FIG. 1, this medium may belong to the computer systemitself. However, the medium also may be removed from the computersystem. For example, security application 106 may be stored in memory136 that is physically located in a location different from processor108. Processor 108 should be coupled to memory 136. This could beaccomplished in a client-server system, or alternatively via aconnection to another computer via modems and analog lines, or digitalinterfaces and a digital carrier line.

More specifically, in one embodiment, host computer system 102 and/orserver computer system 130 is a portable computer, a workstation, atwo-way pager, a cellular telephone, a digital wireless telephone, apersonal digital assistant, a server computer, an Internet appliance, orany other device that includes components that can execute the wormblocking functionality in accordance with at least one of theembodiments as described herein. Similarly, in another embodiment, hostcomputer system 102 and/or server computer system 130 is comprised ofmultiple different computers, wireless devices, cellular telephones,digital telephones, two-way pagers, or personal digital assistants,server computers, or any desired combination of these devices that areinterconnected to perform, the methods as described herein.

In view of this disclosure, the security functionality in accordancewith one embodiment of present invention can be implemented in a widevariety of computer system configurations. In addition, the securityfunctionality could be stored as different modules in memories ofdifferent devices.

For example, security application 106 could initially be stored inserver computer system 130, and then as necessary, a portion of securityapplication 106 could be transferred to host computer system 102 andexecuted on host computer system 102. Consequently, part of the securityfunctionality would be executed on processor 134 of server computersystem 130, and another part would be executed on processor 108 of hostcomputer system 102. In view of this disclosure, those of skill in theart can implement various embodiments of the present invention in awide-variety of physical hardware configurations using an operatingsystem and computer programming language of interest to the user.

In yet another embodiment, security application 106 is stored in memory136 of server computer system 130. Security application 106 istransferred over network 126 to memory 114 in host computer system 102.In this embodiment, network interface 138 and I/O interface 110 wouldinclude analog modems, digital modems, or a network interface card. Ifmodems are used, network 126 includes a communications network, andsecurity application 106 is downloaded via the communications network.

This disclosure provides exemplary embodiments of the present invention.The scope of the present invention is not limited by these exemplaryembodiments. Numerous variations, whether explicitly provided by thespecification or implied by the specification or not, may be implementedby one of skill in the art in view of this disclosure.

1. A method comprising: establishing a break upon entry into one or morespecified system calls of a process running in user space, wherein saidbreak traps said one or more specified system calls in kernel spaceprior to execution; upon said break, dynamically reading informationassociated with said process; and performing a security analysis on saidinformation to determine whether malicious code activity is detected. 2.The method of claim 1, further comprising: wherein upon a determinationthat said malicious code activity is detected, taking protective action.3. The method of claim 1, further comprising: wherein upon adetermination that said malicious code activity is not detected,restarting said system call.
 4. The method of claim 1, furthercomprising: establishing a poll on said process.
 5. A method comprising:opening a /proc file for a specified process that is running on acomputer system in user space; modifying a ctl file of said /proc fileto break on one or more specified system calls of said specifiedprocess; establishing a poll on said /proc file, said poll alerting whenone or more poll conditions are set; upon said break when said pollconditions are set, dynamically reading information associated with saidspecified process; and determining whether malicious code activity isdetected based upon a security analysis of said information.
 6. Themethod of claim 5, further comprising: wherein upon a determination thatsaid malicious code activity is detected, taking protective action. 7.The method of claim 6, further comprising: providing a notification ofsaid protective action.
 8. The method of claim 5, further comprising:wherein upon a determination that said malicious code activity is notdetected, restarting said specified system call.
 9. The method of claim5, wherein said information is read from a /proc status file.
 10. Themethod of claim 5, wherein said information is read from a /proc infofile.
 11. A system comprising: means for opening a /proc file for aspecified process that is running on a computer system in user space;means for modifying a ctl file of said /proc file to break on one ormore specified system calls of said specified process; means fordynamically reading information associated with said specified process;and means for determining whether malicious code activity is detectedbased upon a security analysis of said information.
 12. The system ofclaim 11, further comprising: means for establishing a poll on said/proc file.
 13. The system of claim 11, further comprising: means fortaking protective action.
 14. The system of claim 13, furthercomprising: means for providing a notification of said protectiveaction.
 15. The system of claim 11, further comprising: means forrestarting said specified system call.